The Internet of Things (IoT) allows for the internetworking of devices to exchange data among themselves to enable sophisticated functionality. For example, devices configured for home automation can exchange data to allow for the control and automation of lighting, air conditioning systems, security, etc. In the smart home environment, this can also include home assistant devices providing an intelligent personal assistant to respond to speech. For example, a home assistant device can include a microphone array to receive voice input and provide the corresponding voice data to a server for analysis to provide an answer to a question asked by a user. The server can provide that answer to the home assistant device, which can provide the answer as voice output using a speaker. As another example, the user can provide a voice command to the home assistant device to control another device in the home, for example, a command to turn a light bulb on or off. As such, the user and the home assistant device can interact with each other using voice, and the interaction can be supplemented by a server outside of the home providing the answers. However, homes can have different users interacting with the home assistant device within different contextual environments (e.g., from different locations and at different times) within the home.